Deliverable 5.2 : report on human-machine distributed situation awareness

Häusler Hermann, Ruth; Vetter, Celina; Samardžić, Kristina; Antolović, Dorea; Tukarić, Ivan; Roth, Manuel; Tensfeldt, Luca; Bazina, Mia; Burkhalter, Jennifer

doi:10.21256/zhaw-27264

Please use this identifier to cite or link to this item: https://doi.org/10.21256/zhaw-27264

Publication type:	Working paper – expertise – study
Title:	Deliverable 5.2 : report on human-machine distributed situation awareness
Authors:	Häusler Hermann, Ruth Vetter, Celina Samardžić, Kristina Antolović, Dorea Tukarić, Ivan Roth, Manuel Tensfeldt, Luca Bazina, Mia Burkhalter, Jennifer
et. al:	No
DOI:	10.21256/zhaw-27264
Extent:	176
Issue Date:	2022
Publisher / Ed. Institution:	AISA Consortium
Publisher / Ed. Institution:	Zagreb
Language:	English
Subjects:	AI; Situation awareness; Knowledge graph; Air traffic control; Monitoring task; Eye-tracking
Subject (DDC):	380: Transportation
Abstract:	This report presents the results of two simulation experiments performed with an AI-based situation awareness system (AI SA system) developed in the AISA project to check the accuracy of the AI SA system’s estimations and predictions and its capability to contribute to human-machine team situation awareness. It represents the AISA project deliverable 5.2 Report on Human-Machine Distributed Situation Awareness and contains four topical sections – described below – that address requirements to fulfil the project tasks 5.1 Comparison of SA between AI and ATCO and task 5.3 Human performance in distributed SA. The task 5.2 Risk assessment of AISA is covered in a separate deliverable D5.1 Risk assessment report. • Topical section 1: Measurement of ATCO situation awareness and scanning behaviour • Topical section 2: Comparison of human and machine situation awareness • Topical section 3: Exploration of human-machine team situation awareness and its impact on human performance • Topical section 4: Accuracy of AI SA system’s estimations and predictions and its level of situation awareness Two simulations were conducted with licensed Air Traffic Controllers working as radar executive. Situation awareness was assessed with multiple methods. The probe technique was applied to compare compare human and artificial situation awareness. ATCOs’ experience with AI-based machine situation awareness (receiving “AI SA inputs”) and its impact on performance were explored. Post hoc simulations with data collected in experiment 1 were conducted to assess the accuracy of AI SA systems’ estimations and predictions. Main findings per topical section are: 1. ATCOs with preserved situation awareness have characteristic scanning behaviour: Their gaze is less fixed on aircraft or conflicts, and they filter out more effectively non-critical information than ATCOs with degraded situation awareness do. 2. Partial agreement of human and machine situation awareness on conflict detection. Both human and AI SA system missed conflicts (false negative) and named conflicts that were not present (false positive). The AI SA system is better at monitoring non-obvious/unexpected aspects (e.g., non-conformances). 3. ATCOs detected some conflicts earlier and solved them faster when they received AI SA inputs compared to working without AI SA inputs. Input modality (oral messages) was inadequate due to distraction and additional workload. 4. Successful automation of 46 out of 57 en-route air traffic monitoring tasks. Accuracy of Machine Learning module predictions for CD tested (70%): Partly results were inaccurate, and predictions were partly inconsistent. Plausibility checks on CD module’s inputs and outputs were successful. Limitations reduce the validity of situation awareness measurement for ATCOs (use of an unfamiliar simulation tool), the significance of the results (exploration of human-machine team situation awareness was done with early-stage implementation of the AI SA system and with inadequate design of HMI inducing additional workload on ATCOs). The results generally support the proof-of-concept system of the AISA project in its ability to accomplish en-route air traffic management tasks. Further improvement of accuracy is needed for machine learning modules. Accuracy per se is not sufficient, considerable effort needs to be spent on solutions on how to integrate machine situation awareness. A long anticipation span is desirable for optimisation but does not comply with ATCOs’ need for prioritization of tasks and information. The HMI of the future AI SA system will need distinctive ways of informing ATCOs about aspects of higher or lower urgency. Half of the participating ATCOs were willing to trust future AI-based tools – even after partially unfavourable experiences with an AI-based SA system in the experiment, about one third is neutral and a fifth is negative about including AI in tools.
URI:	https://aisa-project.eu/downloads/AISA_D5.2.pdf https://digitalcollection.zhaw.ch/handle/11475/27264
License (according to publishing contract):	Not specified
Departement:	School of Engineering
Organisational Unit:	Centre for Aviation (ZAV)
Published as part of the ZHAW project:	AISA – Artificial Situational Awareness Foundation for Advancing Automation
Appears in collections:	Publikationen School of Engineering

Files in This Item:

File	Description	Size	Format
2022_Haeusler-Hermann-etal_Deliverable-5-2-report-human-machine-situation-awareness_sesa.pdf		4.39 MB	Adobe PDF	View/Open

Show full item record

Häusler Hermann, R., Vetter, C., Samardžić, K., Antolović, D., Tukarić, I., Roth, M., Tensfeldt, L., Bazina, M., & Burkhalter, J. (2022). Deliverable 5.2 : report on human-machine distributed situation awareness. AISA Consortium. https://doi.org/10.21256/zhaw-27264

Häusler Hermann, R. et al. (2022) Deliverable 5.2 : report on human-machine distributed situation awareness. Zagreb: AISA Consortium. Available at: https://doi.org/10.21256/zhaw-27264.

R. Häusler Hermann et al., “Deliverable 5.2 : report on human-machine distributed situation awareness,” AISA Consortium, Zagreb, 2022. doi: 10.21256/zhaw-27264.

HÄUSLER HERMANN, Ruth, Celina VETTER, Kristina SAMARDŽIĆ, Dorea ANTOLOVIĆ, Ivan TUKARIĆ, Manuel ROTH, Luca TENSFELDT, Mia BAZINA und Jennifer BURKHALTER, 2022. Deliverable 5.2 : report on human-machine distributed situation awareness [online]. Zagreb: AISA Consortium. Verfügbar unter: https://aisa-project.eu/downloads/AISA_D5.2.pdf

Häusler Hermann, Ruth, Celina Vetter, Kristina Samardžić, Dorea Antolović, Ivan Tukarić, Manuel Roth, Luca Tensfeldt, Mia Bazina, and Jennifer Burkhalter. 2022. “Deliverable 5.2 : Report on Human-Machine Distributed Situation Awareness.” Zagreb: AISA Consortium. https://doi.org/10.21256/zhaw-27264.

Häusler Hermann, Ruth, et al. Deliverable 5.2 : Report on Human-Machine Distributed Situation Awareness. AISA Consortium, 2022, https://doi.org/10.21256/zhaw-27264.